Skip to content

Conversation

@sungshik
Copy link
Contributor

@sungshik sungshik commented Feb 26, 2025

This PR makes the following changes:

  • It introduces overflow policies that end-users of the library can use to configure how overflow events should be auto-handled. For now, three overflow policies are declared: NONE (do nothing; the user-defined event handler takes care of it), ALL (larger over-approximation, smaller memory footprint), and DIRTY (smaller over-approximation, larger memory footprint). See the documentation of OnOverflow for details.
  • It provides implementations of NONE and ALL. The next PRs will provide an implementation of DIRTY.
  • It enables the usage of overflow policies for the PATH_AND_CHILDREN watch scope. The next PRs will enable it for the other watch scopes.

The general design of enabling an overflow policy is captured in the following code fragment:

var h = overflowEventHandler().andThen(userDefinedEventHandler());
switch (scope) {
case PATH_AND_CHILDREN: {
var result = new JDKDirectoryWatch(path, executor, h);

That is, the actual event handler passed to the JDKDirectoryWatch constructor is composed of two separate event handlers:

  1. an overflow event handler that generates new CREATED/MODIFIED events (depending on which overflow policy is active);
  2. the user-defined handler that was passed to the constructor of Watcher by the end-user (with a small optimization to ignore overflow events when they're already auto-handled by the policy).

@codecov
Copy link

codecov bot commented Feb 26, 2025

Codecov Report

Attention: Patch coverage is 74.57627% with 15 lines in your changes missing coverage. Please review.

Project coverage is 79.7%. Comparing base (a712a3a) to head (9e71ad7).
Report is 20 commits behind head on improved-overflow-support-main.

Files with missing lines Patch % Lines
...swat/watch/impl/overflows/MemorylessRescanner.java 77.5% 6 Missing and 3 partials ⚠️
src/main/java/engineering/swat/watch/Watcher.java 75.0% 2 Missing and 1 partial ⚠️
...neering/swat/watch/impl/jdk/JDKDirectoryWatch.java 0.0% 0 Missing and 1 partial ⚠️
.../engineering/swat/watch/impl/jdk/JDKFileWatch.java 0.0% 1 Missing ⚠️
...wat/watch/impl/jdk/JDKRecursiveDirectoryWatch.java 0.0% 1 Missing ⚠️
Additional details and impacted files
@@                        Coverage Diff                         @@
##             improved-overflow-support-main     #24     +/-   ##
==================================================================
+ Coverage                              78.7%   79.7%   +0.9%     
- Complexity                               95     107     +12     
==================================================================
  Files                                    12      14      +2     
  Lines                                   428     483     +55     
  Branches                                 41      46      +5     
==================================================================
+ Hits                                    337     385     +48     
- Misses                                   69      70      +1     
- Partials                                 22      28      +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sungshik sungshik mentioned this pull request Feb 26, 2025
7 tasks
@sungshik sungshik marked this pull request as ready for review February 26, 2025 11:23
@sungshik sungshik requested a review from DavyLandman February 26, 2025 11:27
Copy link
Member

@DavyLandman DavyLandman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is going the direction we discussed, but I have some concerns with the implementation. Mostly small, except for the usage of the Files.walk.

Copy link
Contributor Author

@sungshik sungshik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Davy for the review! I addressed most things, except for the renaming of the enum constants, for which I have another suggestion. Once we converged on those names, I'll also update the top-level docs.

@sungshik sungshik requested a review from DavyLandman March 3, 2025 08:16
case NO_RESCANS:
return eventHandler;
case MEMORYLESS_RESCANS:
return new MemorylessRescanner(executor).andThen(eventHandler);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my mind, we should first forward the overflow event, and after that, start the simulation. so a user might be able to recognize simulated events?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this could be useful, but it seems quite nontrivial to implement (but I'm maybe overthinking/misunderstanding the idea). Two issues:

  • The generation of synthetic events runs asynchronously on the executor, similar to how it currently works in JDKRecursiveDirectoryWatcher. I think that's in principle a reasonable choice. However, it could mean that between the original overflow event and the synthetic events, concurrent/unrelated "actual" events come in.
  • Even if there's no interference, there's currently no way of telling how many synthetic events are to be expected.

If we want to support this, then a more reliable approach could be to tag WatchEvent objects as ORIGINAL or SYNTHETIC(i), where i is some kind of an overflow identifier (to be able to distinguish incoming synthetic events that belong to different overflow events, e.g., when they happen shortly after each other). For now, I propose to make a GI issue for this to remember, but not yet implement it, because: (a) it's a cool feature but unclear how much needed; (b) there's already a workaround available, namely, for the programmer to just implement their own overflow handler. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant: chain the simulator after the forwarder, instead of now, first sinulate then forward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The chain consists of:

  • MemorylessRescanner: Asynchronously simulates CREATED/MODIFIED events and forwards them to eventHandler
  • eventHandler: User-defined event-handler that does the actual event-handling work

Changing the order isn't needed to achieve the intended effect (also covered by this test).

Copy link
Member

@DavyLandman DavyLandman Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm a bit confused. I mean that from a client perspective I want us to make sure that on an overflow, the overflow events get send to the client first, and only after that is send do we start sending simulated events.

And maybe this is happening already? But when reading the code I thought: should it not be:

return eventHandler.andThen(new MemorylessRescanner(executor))

but I have to admit I'm missing a bit of context of the eventHandler code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok, that's what you intend to achieve 🙂! Yes, we can enforce that, but as a client, I don't think it makes a difference, because it can't distinguish simulated events from regular events (see also previous comment), regardless of whether the overflow comes first, last, or anywhere in between. I don't see any harm in your proposed change either, though, so now that I understand the intension behind it, I'll change it and merge.

Copy link
Member

@DavyLandman DavyLandman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One final remark left (on the order of the overflow & the syntatic events). But LGRM

@sungshik sungshik changed the title Improved overflow support: Overflow policies Improved overflow support: Overflow auto-handling for directory watches Mar 5, 2025
@sungshik sungshik changed the title Improved overflow support: Overflow auto-handling for directory watches Improved overflow support: Overflow auto-handling for directory watches (non-recursive) Mar 5, 2025
@sungshik sungshik merged commit 546fbe8 into improved-overflow-support-main Mar 7, 2025
13 checks passed
@sungshik sungshik deleted the improved-overflow-support/first-overflow-policy branch March 7, 2025 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants